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Crowdsourced software testing (CST) is an emerging trend in software testing. Compa¬ 
nies and developers assign testing tasks through CST platforms to thousands of online 
testers. Currently, CST platform managers are trying to find and resolve challenges in 
order to reach the best CST practice. Many features have been applied by CST platforms 
to improve CST activities, including notification emails, online chat rooms, forums and, 
most importantly, a CST platform dashboard to view all testing projects and tasks; these 
features have enabled CST to operate efficiently. Still, CST users find it difficult to stay 
abreast of test project updates, maintain their motivation, and avoid frustration. This 
aligns with the increasing studies in the literature that call for more efficient solutions 
to support CST process. This research aims to support CST by searching for potential 
process limitations and overcoming them. In order to do so, a five-stage approach is used: 
first, the current process of CST is investigated through reviewing 15 CST platforms. 
Second, the review has resulted in identifying eight possible activities to improve CST; 
among them six have been selected after a survey distributed to 30 domain experts in 
Stage 3. In Stage 4, we have designed and implemented five process models in a web-based 
system that can fulfill the requirements of the six activities identified earlier. In Stage 
5, we have evaluated these process models through interviews with representatives of 
two CST platforms and an expert tester, and through making scenario-based evaluation 
with 20 domain experts who used the system and rated the value of the processes. The 
results show that the new improvements are sound and can strengthen the CST practice. 

Keywords : Crowdsourcing; software testing; crowdsourced software testing; process 
model; platform. 


1. Introduction 

A software development cycle is not complete without a quality assurance 
phase. The main goal of quality assurance is to ensure that products fulfill or 
exceed customer expectationsP Over the years, many quality assurance methods, 
strategies, and principles have emerged in the software development field such as 
unit testing, functional testing, system testing, mobile testing, etcP 
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Software development practitioners have struggled with how to achieve the best 
quality of testing within a short amount of time and at an efficient cost. Software 
organizations have two test implementation strategies one is having an in-house 
quality department with high-cost employees, equipment, and maintenance, and 
the second is outsourcing quality tasks to another entity, mostly overseas; however, 
this solution is also expensive and complicated, with contracts and great expense 
of time and effort. Both strategies are restricted to the knowledge of a small set of 
testers and thus are limited in terms of quality and efficiency. Furthermore, both 
strategies fail to give a clear view of the intended end user’s vision and expectation 
of the software.® 

With the emergence of crowdsourcing phenomena, software developers and 
investors have utilized it to offer crowdsourced software development testing solu¬ 
tions.® Crowdsourced software testing (CST) overcomes many of the challenges 
of the old approaches, working with the crowd through Web 2.0 technology that 
offers a more efficient distributed online problem solving and production model, 
promising the benefits of a scalable, global workforce while lowering the cost of 
execution.® 

Crowdsourced software testing by nature has testers and clients spread all over 
the world during the software quality assurance phase and communicating through 
testing platforms.® This has driven CST providers to employ coordination tech¬ 
niques to enable testing processes and improve results.® In crowdsourced projects, 
teams that coordinate better are more likely to perform better,® hence support¬ 
ing coordination in CST activities will help avoid delays, save time, and raise test 
team awareness.® Currently, CST platforms are trying to find and resolve coordi¬ 
nation challenges in order to reach the best CST practice. Many features have been 
used by CST platforms to coordinate CST activities, including notification emails, 
online chat rooms, forums and, most importantly, a CST platform dashboard to 
view all testing projects and tasks; these features have enabled CST to operate effi¬ 
ciently.®^ Still, CST users find it difficult to stay abreast of test project updates, 
maintain their motivation, and avoid frustration?3 This aligns with the increas¬ 
ing studies in the literature that call for more efficient solutions to support CST 
process.®^ 

To overcome CST coordination challenges, this research aims to support the 
effective management of CST activities through developing a computer-based sys¬ 
tem. In order to do so, a five-stage approach is used: first, the current process of 
CST is investigated through using the approach of studying coordination depen¬ 
dencies in 15 CST platforms. Second, the review has resulted in identifying eight 
possible activities to improve CST; among them six have been selected after a sur¬ 
vey distributed to 30 domain experts in Stage 3. In Stage 4, we have designed 
and implemented five process models in a web-based system that can fulfill the 
requirements of the six activities identified earlier. In Stage 5, we have evaluated 
these process models through interviews with representatives of two CST plat¬ 
forms and an expert tester, and through making scenario-based evaluation with 
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20 domain experts who used the system and rated the value of the processes. The 
results show that the new improvements are sound and can strengthen the CST 
practice. 

This paper is divided as follows: Section [2] provides background about CST 
while the related work is discussed in Sec. OH Section 0] describes the approach while 
Secs. |5H9] present the five stages of this research. Finally, the paper is concluded in 
Sec.HOl 


2. Background 

2.1. Crowdsourced software testing (CST) 

Crowdsourced software testing is an emerging trend in software testing. Compa¬ 
nies and developers assign testing tasks through CST platforms to thousands of 
online testers. Workers are thus provided with more job opportunities and extra 
income, and employers are provided with fast, high-quality services at low costs. 
Leicht et aZP describe crowdsourced software testing as “a specific application of 
crowdsourcing in the domain of software development. It refers to the outsourcing 
of software testing activities to the crowd”. 

There are many advantages to crowdsourced testing when compared to tradi¬ 
tional, lab-based testingP Advantages include the easy and fast method of recruit¬ 
ing real end users as participants (testers) through crowdsourcing platforms rather 
than traditionally asking people (end users) to come to the test lab and participate 
in usability testing. The duration of the whole testing process in lab testing can 
take weeks, especially if the customer/user is at a different site with the addition of 
traveling, greeting, and setting up time, while in crowdsourced testing, the whole 
process can be done in hours. Moreover, the total cost of crowdsourced testing 
is significantly lower due to savings in tester payments, facilitators, travel, labs, 
and equipment. Hence, the low-cost testing allows for more test iterations, more 
test improvements, and more testers. Finally, crowdsourced testing participants are 
from all over the world, which makes it very easy to conduct tests with participants 
from various backgrounds. This is especially helpful for international websites and 
applications. 

Many researchers, including Brabham^ HossainJ^ and Schenk and Guit- 
tar d — 1 agree that there are three main components to any CST system: 

• Crowd requester: the user or company with the need to test a product such as a 
web application or a smartphone application. The crowd requester shall submit 
his project files/links and project documentation as needed. 

• Crowd tester (worker): online users looking to utilize their time and gain some 
benefits (financial, enhancing testing skills, or building a reputation) by testing 
software, websites, and applications, and then submitting issues and bugs. In 
crowdtesting, online users are required to register on a CST platform in order to 
join its community as testers. 
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• Crowdtesting platform: an online portal with testing services that enables any 
Internet user to become a tester and enables products’ owners to test their prod¬ 
ucts using a fast, low-cost, international testing method. 

Currently, CST platforms have built a CST business model that uses the following 

actor^rjT5f 


• Testers: random online users interested in improving their testing skills and 
possibly gaining some financial benefits. Testers perform testing activities on 
software/applications, and then submit their findings (bugs) through the CST 
platforms. They are normally diverse in their demographics and experience. 

• Project manager (PM): an employee of the testing platform responsible for man¬ 
aging the crowds and test projects, as well as building test teams. Most impor¬ 
tantly, he manages the test cycles and maintains the relationship between clients 
and testers. 

• Test team lead (TTL): an optional role that adds an extra layer of quality control, 
a member of the crowd with a good testing record (high score/rank) responsible 
for the initial review of bugs in the software. 

• Client: a client or project owner who crowdsourced the testing activities to the 
crowd. The client is responsible for submitting test projects files/links and doc¬ 
umentation, as well as reviewing test reports for validation purposes. 

2.2. The concept of coordination 

Coordination is an essential aspect of software development. Mintzber^®state that: 
“every organized human activity — from the making of pottery to the placing of a 
man on the moon — gives rise to two fundamental and opposing requirements: the 
division of labor into various tasks to be performed and the coordination of those 
tasks to accomplish the activity”. 

Sufficient coordination among the activities involved in the software develop¬ 
ment cycle is required in order to achieve a successful software system. However, 
this coordination is hard to achieve, especially when project complexity increases 
because of team distribution.® 

Malone and CrowstorP^ have developed a general coordination theory, which 
considers coordination definitions from a variety of fields such as science, psychology, 
computer science, and economics. Later, they provided the following definition® 
“Coordination is managing dependencies between activities ” (p. 6). 

As the definition illustrates, dependencies require coordination, and here depen¬ 
dencies are between activities. Coordination dependencies are characterized mainly 
in three basic types, which are fit, flow, and sharing. 90 Fit dependency rises when 
multiple activities produce or use the same resource; flow dependency rises when 
one activity produces a resource to be used by another activity while sharing depen¬ 
dency rises when one resource is used by multiple activities. 


2040009-4 





Int. J. Coop. Info. Syst. Downloaded from www.worldscientific.com 

by AUCKLAND UNIVERSITY OF TECHNOLOGY on 05/25/20. Re-use and distribution is strictly not permitted, except for Open Access articles. 


Towards Better Crowdsourced Software Testing Process 


In the context of managing CST activities, all the three dependency types exist, 
for instance: 

• Fit Dependency: This kind of dependency appears when many testers collectively 
work to produce the final test report in a test cycle. Moreover, several test cycles 
produced by several test team members can contribute to developing a specific 
test project. 

• Flow Dependency: The CST process consists of a series of sequential steps in 
order to achieve the final test results. Test team members often need artifacts 
that are the result of the activity from a previous step in the testing process. An 
example is the test reports validation process, in which the PM cannot validate 
test reports until testers submit them. Moreover, in the feedback section, the 
PM can leave comments on specific bugs and wait for the tester’s reply for more 
information. 

• Sharing Dependency: The CST platforms use a shared testing environment with 
crowd testers where all testing activities take place. Another example is that the 
crowd testers share the same test cycle time to perform testing activities. 

Understanding the concept of coordination will be used later in this research to 
study coordination dependencies involved in the process of crowdsourced software 
testing and suggest process improvements (more details on this are provided in 
Sec. |6]). 


3. Related Work 

The crowdsourced software testing field is an emerging area with computer plat¬ 
forms rapidly increasing in the last few years. The current literature aims mainly to 
assess the value of CST (Table [I}. Nebeling et all ^introduce a crowdsourced testing 
toolkit called CrowdStudy. CrowdStudy conducts website usability testing as a ser¬ 
vice by selecting testers through simple integration with a crowdsourcing platform 
(mechanical truck, in this case); gathering usability information such as number 
of clicks, navigation, etc.; analyzing it; and finally providing valuable insights and 
recommendations based on how users interact, preferable device orientation and 
font size, etc. 

Liu et al 7 perform test evaluation for a graduate school website. They explore 
crowdsourcing as an alternative way to conduct remote usability testing. They 
found that while the quality and user feedback are not as good as in lab testing, 
they could be enhanced by careful design of tasks and surveys and by conducting 
many test rounds, as crowdsourcing is faster, cheaper, and easier to perform with 
participants from diverse backgrounds. In Refs. 71 and IT8l the authors pointed out 
that testers do not give many additional comments (in written form) compared to 
lab testers. This can be a negative aspect, as the message may not be delivered 
well. 
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Table 1. Research studies investigating CST. 


Paper 

Assessment Approach 

CST Platform 

Identified Challenges 

17 

Experimental 

New tool integrated 
with MTurk 

None 

7 

Experimental 

MTurk and 
CrowdFlower 

Quality of CST results 

Clarity of reports 

The need for multiple rounds 
of testing 

No mechanisms to 
prevent spammers 

18 

Experimental 

New tool integrated 
with MTurk 

Clarity of reports 

19 

Experimental 

MTurk 

None 

20 

Experimental 

None 

The benefit does not scale 
with the cost 

21 

Experimental 

None 

Complexity of testing tasks 

22 

Interviews/questionnaire 

None 

Time pressure 

Competition pressure 

23 

Experimental 

Not mentioned 

None 

4 

Interviews 

testCloud 

Quality of results 

Manual process of 
selecting testers 

Lack of mechanisms to define 
testing requirements 
with customers 

Involvement of client in the 
testing process 

Manual process of receiving 
testing requirements 
from clients 


Dolstra et al P 9 utilize automated crowdsourced testing for GUI tests, which are 
hard to create and maintain. They describe a method for crowdsourcing GUI tests 
based on installing the tested system in virtual machines that are then presented to 
a geographically dispersed pool of workers. Their several experiments on Amazon 
mechanical truck demonstrate that their approach works well in terms of technical 
feasibility. 

Shinsel et alW^ assessed the value of engaging a small number of end users for 
the testing of an intelligent assistant from a cost-benefit perspective. They found 
that a mini-crowd provides several advantages besides the decrease in workload. 
Nevertheless, the advantages did not scale linearly as the crowd size increases; 
there is a point of diminishing returns. 

Malone and Dunnd^ compared between the types of defects users find in the 
field and in-house test teams. The results show that security and performance bugs 
are best handled by using traditional in-house teams. Moreover, customers are 
skilled at finding normal functional bugs. 

Guaiani and Muccini^ studied the possibility of integrating crowd-based testing 
with traditional lab-based testing. The crowd were directly asked about the main 
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challenges they face during the testing. Competition pressure and time pressure 
were among the main identified challenges. 

Leicht et alW I discuss the organizational cost-benefit perspective of crowdsourc¬ 
ing and conduct two case studies to address the question of when crowdsourcing is 
advantageous. They select the domain of crowdsourced software testing to test their 
proposed theory, analyzing the behavior of two organizations performing CST and 
traditional in-house testing for the same mobile application. They compare the two 
approaches in term of effectiveness and costs of testing. The result is that crowd¬ 
sourced testing provides significant advantages in terms of speed, heterogeneity of 
testers, and user feedback as added value. 

Zogaj et alW perform a case study involving a German start-up crowdsourcing 
company called testCloud that offers software-testing services for entities planning 
to partly or fully outsource their testing activities. The case study shows that test- 
Cloud’s challenges are managing the process, managing the crowd, and managing 
the technology. Even though valuable insights are presented, the study discusses 
the challenges facing only one CST platform. Managing the process challenges are: 
finding appropriate mechanisms to define testing requirements with customers, the 
potential of receiving poor quality output from testers, and the difficulty of involving 
customers in the process. The challenge associated with managing the crowd are: 
the manual allocation of testing jobs. Further, managing the technology involves 
the absence of computer solutions to facilitate receiving testing requirements from 
customers. 

Our work differs from the previous research studies from different angles. First, 
the previous work identifies the challenges by examining either one platform or by 
comparing the whole process of CST to the traditional testing. In our work, we focus 
on investigating the current process of CST through reviewing 15 CST platforms. 
Hence, it provides more holistic view of the challenges facing CST. Second, unlike 
our work, the previous work aims to evaluate the CST without providing solutions 
to the identified challenges. 


4. Approach 

The aim of our research is to investigate the current process of CST practice and to 
search for possible advancements to the process. To achieve this aim, the research 
is divided into a set of stages as follows: 

Stage 1: Investigating the current process of CST : We conducted comprehensive 
investigation of 15 CST platforms. The result of the review is presented in Ref. |2T| 

Stage 2: Identifying potential advancements : Based on our review in Stage 1, we 
propose in this stage a set of improvements that we believe it can possibly enhance 
CST. 

Stage 3: Evaluation of the proposed support: The proposed activities are evaluated 
using a survey distributed to 30 domain experts. 
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Stage 4: Design and implementation : The activities that are agreed on are designed 
and implemented at this stage. 

Stage 5: Final evaluation : In this stage, the implemented features are evalu¬ 
ated using in-depth interviews and scenario-based evaluation with 20 domain 
experts. 


5. Stage 1: Investigating the Current Process of CST 

In order to understand the CST process, an investigation of current platforms was 
conducted in Ref. [23J A search of current CST providers was conducted. The study 
focuses only on platforms built mainly for software testing and, hence, general- 
purpose crowdsourced platforms are not covered. The procedure we used to select 
the platforms are as follows: 

• Step 1: Creating a pool for the existing CST platforms: The platforms have been 
added from three resources: first, the researchers’ previous knowledge about CST 
platforms and second, searching the Internet especially key research and soft¬ 
ware testing websites such as Refs. [2]2 and [26] Third, we have searched also 
into the popular academic databases to see if there are any platforms devel¬ 
oped in the academia (namely: ScienceDirect, IEEE Xplore, Web of Science, 
ACM Digital Library, SpringerLink and Scopus). We attempted a tremendous 
number of strings. However, the main strings that yielded good results are as 
follows: 

o (‘Software Testing’ OR ‘Software Validation’ OR ‘Software Verification’) 
AND (‘Crowdsourcing’ OR ‘Crowdsourced’ OR ‘Crowd’) AND (“Tool” OR 
“Platform” OR “System”), 

o (‘Crowdtesting’ OR ‘Crowdsourced Testing’) AND (“Tool” OR “Platform” 
OR “System”). 

This step resulted in finding 34 CST platforms. 

• Step 2: The researchers removed the platforms that they were not able to study 
their process at the time of investigation [September-November 2017] (i.e. due to 
the lack of sufficient resources such as documentations, demos, etc.). Six platforms 
have been removed as a result of this step. 

• Step 3: The list of platforms is ordered based on the available data about the 
most influencing platforms. Many factors have been considered including the 
published number of users, academic surveys about crowdsourced software testing 
such as Refs. and [28] and recommendation of the independent research and 
software testing websites such as Refs. [25] and [26| As a result, the researchers 
selected fifteen platforms (Table [2]) for investigation. 

To ensure that all the available activities are investigated, three cycles of 
review and refinement were carried out by both researchers. After each cycle, 
both researchers met to discuss the results. The review is conducted using different 
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Table 2. Selected CST platforms. 


CST Platforms 

Head 

Quarter 

Year of 
Foundation 

No. of 

Testers 

Testing Services 

uTest 

USA 

2007 

250,000 

Functional Testing, 
Usability Testing, 
Security Testing, 
Accessibility Testing, 
Test Automation 

Passbrains 

USA 

2012 

50,000 

Functional Testing, 
Usability Testing, 
Security Testing, 
Performance Testing, 
Localization Testing 

Crowdsprint 

Australia 

2014 

100,000 

Functional Testing, 
Usability Testing, 
Security Testing, 
Regression Testing, 
Cross-Browser Testing, 
Compatibility Testing, 
Localization Testing 

Test IO 

Germany 

2011 

50,000 

Functional Testing, 
Usability Testing 

Usability Hub 

(changed recently 
to User Crowd) 

Australia 

2008 

100,000 

Usability Testing 

BugFinders 

UK 

2011 

55,000 

Functional Testing, 
Usability Testing, 
Security Testing, 
Accessibility Testing, 
Localization Testing 

Crowdsourced 

Testing 

Canada 

2011 

74,000 

Functional Testing, 
Usability Testing, 
Localization Testing 

UserTesting 

USA 

2007 

Not mentioned 

Usability Testing 

Rainforest 

USA 

2012 

50,000 

Functional Testing, 
Cross Browser Testing 

99tests 

India 

2011 

30,000 

Functional Testing, 
Usability Testing, 
Security Testing, 

Test Case Design, 

Test Automation 

MyCrowd QA 

USA 

2013 

50,000 

Usability Testing, 

Cross Browser Testing, 
Compatibility Testing, 
Regression Testing 

Global App Testing 

UK 

2013 

25,000 

Functional Testing, 

Test Case Execution, 
Localization Testing, 
Regression Testing 

Pay4Bugs 

Hong Kong 

2009 

Not Mentioned 

Functional Testing, 
Localization Testing 


( Continued ) 
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Table 2. ( Continued) 


CST Platforms 

Head 

Quarter 

Year of 
Foundation 

No. of 
Testers 

Testing Services 

TestArmy 

Poland 

2010 

7,000 

Functional Testing, 
Security Testing, 
Performance Testing 

Ubertesters 

USA 

2012 

25,000 

Functional Testing, 
Usability Testing, 
Localization Testing 


methods, including: 

• Reading the formal description of available features, which is usually available 
on the CST platform’s website or documentation produced by the company. 

• Watching demos on the platform’s functionalities. 

• Communicating with the platform’s customer support asking about specific fea¬ 
tures provided for clients. 

• Working on free versions of some platforms by participating as a tester and joining 
test cycles, submitting bugs, communicating with project managers/TTL, and 
getting paid for approved bugs. 

• Visiting the forums and reading customer feedback on the CST platform’s 
website. 

• Finally, the discussion of CST process in the papers 8,15,28 have supported the 
understanding of the CST activities. 

It was found that a general process is followed in all CST platforms, which can 
be seen and understood from a simple high-level perspective as appears in Fig. HJ 
This diagram summarizes all the activities found in CST platforms. The process 
can be divided into three phases as follows: 

Submitting project for testing: It starts when client submits a new test project 
request. The request is assessed from a platform coordinator based on the testing 
environment, test type, number of testers and devices, expected number of valid 
defects, and duration. The platform coordinator agrees then with client about a 
payment plan and assigns a project manager based on availability, knowledge of 
testing area, and experience. 

Designing test cycle and performing testing: Client uploads the actual test 
project that is further reviewed by project manager to design test cycle defining 
start and end dates and test team leader (TTL). Afterwards, the project manager 
selects and sends invitations to potential testers based on project test require¬ 
ments, previous experience, test field, test devices, and tester’s demographics and 
score. A crowd tester has the option to accept or reject any test project. Testers 
can view full project details through the tester’s dashboard on their testing plat¬ 
form, including detailed project specifications, scope, payments and guidelines and 
then submit test reports. The CST platform’s reviewers (project manager/TTL) 
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Fig. 1. Crowdsourced software testing general process. 

will select and validate the most interesting bugs until the agreed-upon num¬ 
ber of bugs is reached, and then all reviewed bugs will be sent to the client for 
confirmation. 

Finalizing testing process: The client identifies accepted and rejected bugs, and 
then project manager prepares test report that normally includes defects explored, 
tester feedback, test summary, and recommendations. After the test cycle finishes 
and the project is closed, payment is processed as stated in the project budget 
details. 


6. Stage 2: Identifying Potential Advancements 

The process identified in Fig. [l] is utilized to propose process improvements. We 
used the approach of studying coordination dependencies in order to provide process 
improvement. This approach has been intensively used in the literature pSBES] 

We followed heuristics provided by Crowston and OsborrPHl for analyzing depen¬ 
dencies and coordination mechanisms in a situation to make dependency-focused 
analysis. They proposed this procedure: identify dependencies, then search for coor¬ 
dination mechanisms. In other words, look for dependencies, then ask which coor¬ 
dination support is used to manage those dependencies. Crowston and OsborrP^ 
suggested asking questions such as the following to discover dependencies and 
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coordination mechanisms: 

• What are the inputs/outputs to each activity (e.g. informational and other nec¬ 
essary pre-conditions / post-conditions). 

• What potential performance problems associated with this process? Do these 
problems reflect unmanaged [or poorly managed] dependencies? 

Each activity involved in the process is examined by both researchers to see how 
it would be performed better. The list of suggested coordination improvements were 
refined many times until both researchers reached an agreement to the list shown 
in Table |3l The list includes only the activities where both researchers believe that 
a possible improvement can be provided by introducing new coordination mecha¬ 
nisms. The proposed coordination support may help accelerating the process and 
improving the process efficiency and quality. This has resulted in new coordination 
mechanisms to the involved activities such as automation, customized notifications 
and reminders that possibly can support the overall process. The details are dis¬ 
cussed in what follows: 

Al. PM assignments 

Clients have to register and select a payment plan that fits their needs before a 
dedicated crowdtesting PM is assigned. After a suitable PM is assigned, the client 
can submit his project for testing. This is a flow dependency. The process of selecting 
a suitable project manager may take time and is subject to human error, as this 
process is manual in all CST platforms. This may be improved by automating PM 
assignments. The system can assign PMs to new test projects based on matching 
their requirements and preferences. 

A2. Announcement alerts 

The client submits all test project files and documentation and the PM must review 
all test project information before posting an announcement to recruit interested 
qualified testers. After the announcement is posted, interested testers can sign up 
and the PM can review their profiles. Current CST platforms (i.e. uTest and Pass- 
brains) that do announce some upcoming test projects maintain an announcements 
page and send newsletter e-mails to all testers. This approach can be annoying, 
with some testers receiving many unwanted e-mails. Additionally, this keeps testers 
regularly checking the website for relevant test project announcements. However, 
this could be further improved by identifying suitable testers for each new test 
project announcement and notifying them to sign up for it. 

A3. Managing testers’ responses to test cycle invitations 

As the crowdsourced test project progresses and after inviting selected suitable 
testers, testers must respond by accepting or declining the invitation. After the 
testers accept their invitations, they can review full test project details to start 
submitting test reports (bugs). Currently, CST platforms have testers log in to 
accept/reject an invitation so they can secure a place on the testing team. Test 


2040009-12 



Int. J. Coop. Info. Syst. Downloaded from www.worldscientific.com 

by AUCKLAND UNIVERSITY OF TECHNOLOGY on 05/25/20. Re-use and distribution is strictly not permitted, except for Open Access articles. 


Towards Better Crowdsourced Software Testing Process 


slots are booked on a first-come, first-served basis. However, there is no support for 
responding via mobile. Further support could be applied through a response link 
in the invitation that allows testers to answer without the need to sign in. 

A4. Managing non-active testers 

Testers must submit test reports to be validated, and at the end of the test cycle, 
the CST platform system will automatically update each tester evaluation score 
based on his performance in this test cycle. Currently, most CST platforms do not 
acknowledge non-active testers. Few CST platforms such as Test 10 and Rainforest 
give a lower rank to non-active testers. This approach may decrease the test quality 
and client trust by having fewer testers work than what was promised. Additionally, 
available testers may lose a chance of working on a paid test project while other 
participating testers may not. Further support can be applied by identifying non¬ 
active testers in the middle of the test cycle and asking them either to actively 
participate or withdraw. If a non-active tester withdraws, other testers shall be 
notified of a new test slot opening. This approach will give testers a chance to 
cancel if they cannot participate and give other testers the chance to participate in 
a paid test cycle. 

A5. Managing technical issues 

Starting a new test cycle and after testers read the full test project description, 
a tester will try to access the testing environment to start testing and submitting 
test reports (bugs). However, some testers may face technical issues accessing the 
test environment, especially with mobile testing. In such cases, CST platforms have 
testers send e-mails to support teams or PMs and wait for a response. However, 
the response could take a long time, which may cause the tester to lose the chance 
to participate and the advantage of viewing the software before other testers find 
all valuable bugs. Further support may be provided by offering a technical sup¬ 
port option in the CST platform. The technical support function allows testers to 
complete and submit a form, which then will notify the assigned PM. The PM 
can review all information and sufficiently respond. In addition to improving the 
response time with technical issues, this approach creates a record for all technical 
issues to which management can return at any time. 

A6. Confirming test reports notification 

Testers have to submit a sufficient amount of test reports, and then the PM will 
validate them, after which the client has to confirm them. After bug evaluation is 
done, the CST system will automatically update the tester’s evaluation score based 
on his performance in this test cycle. The final test report can be submitted to the 
client and testers can receive payment. Currently, CST platforms coordinate this 
dependency by having PMs manually contact clients to inform them of validated 
bugs. However, clients may take time to confirm reported bugs, keeping PMs and 
testers waiting. CST platforms do not instantly notify clients for faster confirmation. 
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This could be improved by automatically notifying clients to confirm bugs as soon 
as the PM evaluates the first bugs. 

A7. Managing test project’ progress 

CST testers and clients may waste time doing unnecessary work as the test project 
progresses. All CST platforms do not have progress notification to testers and 
clients, by which the PM validates the required number of bugs. Automatic sup¬ 
port may be provided through e-mail notification to testers and clients about test 
project progress. 

A8. Managing test project updates 

Testers have to read and understand a full test project description before submitting 
any test reports (bugs). After testers submit a sufficient amount of test reports, the 
bugs’ validation and testers’ evaluation activities can begin. During active test 
cycles, a dependency may exist when some important updates occur with new 
information that may change testers’ course of work. Currently, project updates of 
any kind (e.g. updated scope) are coordinated by emailing them to participating 
testers. In some CST platforms (e.g. uTest, 99tests), updates are also posted by 
PMs in chatrooms. However, testers are not always aware of new e-mails, especially 
when they are working within limited timeframes. It is important for testers to be 
up to date on all test project information in order to submit quality reports. This 
may be improved by providing instant in-application notification for test project 
updates for all testers in the project’s test cycle. 

7. Stage 3: Evaluation of the Proposed Support 

In order to evaluate the need for the proposed support, a survey has been distributed 
to 30 domain experts. The survey was distributed through a link to domain experts 
on CST forums and communities. Additionally, the link was published on Twit¬ 
ter using the following hashtags: ^Crowdsourcedtesting ^crowdsourcing ^testing. 
Private messages to top testers were also sent. 

The survey consists of two parts. The first part inquires about background 
information such as job title, experience, and the CST tool used. The second part 
provides a list of CST-related activities for which the proposed support is suggested 
(Table |3]). The respondents were asked to rate the extent to which they agree 
whether the support is necessary. 

The respondents came from three main backgrounds: Testers, project managers 
and test team leads. Figure [2] illustrates that 90% of the respondents were testers. It 
also shows that most participants use the uTest platform (83%), probably because 
uTest is currently the largest CST platform in the world while most others are much 
smaller companies in comparison. Other platforms include BugFinders platform 
(19%), Test IO (16%), and 99 tests (16%). 

Each activity that affects the CST process as explained before (Sec. 4.2) is 
tested against a defined question. Respondents have been given three choices to 
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Table 3. Support needed for CST. 


No. CST Activity Description 


A1 PM 

assignment 


A2 Announcement 
alerts 


A3 Managing 
testers’ 
responses 
to test cycle 
invitations 


A4 Managing 
non-active 
testers 


A5 Managing 
technical 
issues 


A6 Confirming 

test reports 
notification 


Before test project 
can start and client 
can submit the test 
files, a suitable PM 
must be assigned 

Testers must know 
about upcoming test 
projects of their 
interests in order to 
sign up 


Before testers can 
view test project 
details and start 
testing, testers must 
accept the test 
invitation 

Testers in active test 
cycles must submit 
test reports (bugs) to 
be evaluated and 
receive payment 


Technical issues must 
be resolved so testers 
can work on the 
testing environment 


Before testers receive 
payment, bug 
validation and client 
confirmation must be 
done 


Current Support 

PMs are assigned 
manually, which may 
cause delays or human 
errors 

Testers are informed 
via news pages and 
newsletter e-mails, 
which could be 
un-useful 


Testers must log in to 
platform’s website to 
respond. This may 
cause testers to lose 
their test slots 


Testers who do not 
submit any test reports 
are unnoticed or may be 
given a low evaluation, 
which is inefficient to 
clients and testers 


Technical issues are 
discussed and resolved 
over emails between 
testers and PMs/ 
support team. This 
could cause delays and 
affect the tester’s work 


PM manually informs 
clients to confirm bugs, 
which may cause 
payment delays 


Proposed Support 

Further support can 
be provided by an 
automatic assign¬ 
ment mechanism of 
project managers 

Targeted 

notifications can be 
provided by sending 
a notification e-mail 
to interested testers 
in the announced 
project’s field 

Further support can 
be provided by 
improving the test 
cycle invitation to 
allow testers to 
respond through 
mobiles 

Further support can 
be provided by iden¬ 
tifying non-active 
testers and sending 
a notification e-mail 
to them to fulfill 
their commitment or 
to withdraw and let 
new testers join 

Further support can 
be provided through 
a technical support 
option that allows 
testers to complete 
an issue form, and 
then the system will 
alerts the assigned 
PM. The PM can 
review full details of 
the issue and 
respond. The tester 
will be notified of 
the result 

Further support can 
be provided by 
alerting the client to 
review and confirm 
bugs when the PM 
starts validating 
them 


( Continued) 
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Table 3. 

( Continued ) 


No. 

CST Activity 

Description 

Current Support 

Proposed Support 

A7 

Managing 
test project’s 
progress 

Before testers and 
clients submit/ 
confirm new bugs, 
they must be aware of 
test project progress 

Testers and clients may 
not be aware of test 
project progress, so 
they may submit new 
unnecessary work after 
the PM validates the 
required number of 
bugs for test cycle 

Further support can 
be provided by 
alerting testers and 
clients when the PM 
validates the 
required number of 
bugs for test cycle 

A8 

Managing 
test project 
updates 

Before testers submit 
their bugs, they must 
know accurate test 
project details 

PM sends update 
emails and may post in 
chatrooms when clients 
provide new 
information. Testers 
may not be aware of 
this while working 

Further support can 
be provided by 
instant in¬ 
application 
notification for test 
project updates 



100 % 



Fig. 2. Respondents’ background. 


assess the activities: agree, disagree or undecided. The results are shown in Table 0] 
and Fig. [3j We have applied the 50% Rule to determine which CST activities should 
be supported through developing new proposed models. This rule means that if at 
least 50% of respondents are in the support of an opinion then it is accepted. This 
method has been followed by other research As a result, the activities (Al, 

A3) have been eliminated. 


Table 4. Respondents’ answers. 


No. 

Activity 

Agree 

Disagree 

Undecided 

Al 

Project managers’ assignments 

10 

11 

9 

A2 

Announcement alerts 

22 

3 

5 

A3 

Managing testers’ responses to test cycle invitations 

13 

7 

10 

A4 

Managing non-active testers 

22 

3 

5 

A5 

Managing technical issues 

21 

5 

4 

A6 

Confirming test reports notification 

22 

4 

5 

A7 

Managing test project’s progress 

16 

3 

11 

A8 

Managing test project updates 

20 

5 

5 
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Activities 

Agree ■ Disagree Undecided 

Fig. 3. Histogram of respondents’ answers. 


8. Stage 4: Design and Implementation 
8.1. Design 

This section presents the design and implementation of the proposed CST activities. 
It discusses how a computer-based system would be able to provide the needed 
support identified in Stage 2. 

A set of process models (Table |5]) has been created for the proposed activities 
agreed on in the previous evaluation. These models provide a visual representation 
of how the activities can be developed. The models went through a few cycles before 
they reach the final form. 

The process models were created using business process model and notation 
(BPMN), which demonstrate both the activity flow and how they relate to different 
involved roles (i.e. testers, project managers, and clients). 

PI: Announcements alerts 

This process model covers the proposed activity A2-announcement alerts. The pro¬ 
cess model in Fig. H] illustrates tester’s required profile setup activities to receive 
relevant announcements, while Fig. [5] shows the sequence of activities required when 
a project manager initiates a new project announcement. 

• First, the tester must update his profile to set “announcements alerts” on, then 
select preferred project specifications and “save” this action. 


Table 5. Proposed process models and corresponding activities. 


Process Model 

Corresponding Activities 

PI: Announcements alerts 

A2 

P2: Managing non-active testers 

A4 

P3: Managing technical issues 

A5 

P4: Confirming test report notifications and 
managing test project’s progress 

A6, A7 

P5: Managing test project updates 

A8 
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Fig. 4. Tester’s update personal profile. 



• The system will save these settings for future use, and system checks that using 
this information will take place with every published announcement. 

• The project manager plans a new test project, confirms its preferences, and then 
navigates to the news management page. 

• The system will validate the project manager’s list of authorized projects assigned 
to him, and will then display these projects in a list to associate a project with 
the new announcement. 

• The project manager will select a project, enter a suitable title, and write the 
post. Finally, he will publish the announcement. 

• The system will validate those suitable testers to be automatically notified of 
this announcement based on matching mechanisms between tester and project 
preferences, which are location, test type, language, and domain. If at least one 
preference matches, then the tester will be notified. 
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P2: Managing non-active testers 

The workflow on Fig. |6] demonstrates our proposed solution for activity A4 (man¬ 
aging non-active testers) in an active test cycle. 

• With every created test cycle, the system creates a flag for its middle time. When 
an active test cycle reaches its middle time, the system will search for non-active 
testers. In this research, non-active testers are considered testers who did not 
submit any test reports (bugs). 

• The system will identify non-active testers and notify them to actively partici¬ 
pate. They will be given an additional 15% of test cycle time to reply (threshold 
agreed on during the interviews). If tester does not respond, the system will auto¬ 
matically discard him from the team and notify PM and other accepted testers 
of an open tester slot to join. 

P3: Managing technical issues 

This process model covers the proposed activity A5-managing technical issues. The 

process model shown in Fig. 0 illustrates the steps involved when a tester encounters 

a technical issue. 

• After the tester faces a technical error and cannot access the testing environment, 
the testers will initiate a new technical error message and enter all required 
details (the project with technical error, title, images, description). The sys¬ 
tem will perform a validation on authority to display only projects the tester is 
involved in. 

• The system will automatically notify the assigned project manager of the new 
technical error message. 
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Fig. 7. Rise technical error message by tester. 


• The PM will browse the technical errors list and review new messages, write a 
suitable solution, and close the message. The system will perform a validation on 
authority to display only technical error messages for projects that belong to the 
PM. The system will notify the tester of new solutions. 

P4: Confirming test report notifications and managing test project’s 
progress 

This process model covers activities A6 and A7. The below workflow in Fig. |8] 
discusses the proposed solution of alerting clients to confirm bugs as soon as possible 
and show how we plan to send progress alerts to testers and clients. 

• When the PM starts verifying the submitted test reports by testers, the system 
will monitor PM evaluation and with the first accepted bug, the system will 
automatically send a notification email to the client to start confirming bugs. 

• With every evaluated bug by the PM, the system will perform a validation check 
on the number of required bugs for this test cycle. If the number of required bugs 
is reached, the system will automatically send a notification email to testers and 
client informing them of test cycle progress. 

P5: Managing test project updates 

This workflow (Fig. [9]) addresses activity A8, the importance of instantly informing 
testers about test project updates that may affect their work significantly (e.g. an 
updated scope). 

• When a client changes any of the test project details, he will contact the PM, 
and the PM shall update testers accordingly. The PM will log in to the system 
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Fig. 8. Client confirmation and progress alerts. 



Fig. 9. In-application update notifications. 


and navigate to the project’s details, where he will update required fields and 
finally save this action. 

• The system will perform a validation on authority to display only projects in 
which the PM is involved. The system will identify participating testers in this 
project and send a notification email with new updates. 
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• The system will display an in-application notification to all participating testers 
to update them with new information. 

8.2. Implementation 

As proof of concept, a computer system has been developed to demonstrate the 
possibility to implement the proposed process models. All process models have 
been implemented to ensure system feasibility. 

A web-based system has been developed using PHP and JavaScript, which are 
both well-known languages. The database was developed using MySQL, which 
is compatible with and supported by PHP. The reasons for choosing these lan¬ 
guages are portability and popularity. For brevity, we describe in this section 
the system database along with the implementation of Process 1 (Announcement 
alerts) only. The full implementation details of all process models can be accessed 
through.^ 

System database 

The back-end SQL database of the proposed data models has been created fFig. HQ]) . 
This database is capable of storing the necessary data about all CST process mod¬ 
els provided in this research. It keeps track of the basic information of the main 



Fig. 10. System database. 
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roles: testers, clients and project managers. For testers, the system stores also their 
devices along with projects and test cycles they participated in and whether bugs 
are identified by any of them. Moreover, the system saves the list of clients and 
projects supervised by a project manager and whether bugs submitted by testers 
are verified by clients and project managers (further details about the database can 
be found in Ref. 136). 

Implementation of process model PI: Announcement alerts 

1. The project manager initiates adding new announcement. 

First, the project manager will log in according to his privilege, and options 
will be available as seen in Fig. [TT] (e.g. projects that he is authorized to 
work on). 

After selecting “News”, the “Add new announcement” option will be dis¬ 
played (Fig. [12]). 



Home 

News 

Contact 

Welcome pm name 

Sign out 


My Projects 
My Profile 
Technical Errors 
Change Password 

Fig. 11. PM home page. 



Fig. 12. New announcement page. 
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Home JNTews Contact 


My Proj ects 
My Profile 
Raise Technical Error 
Change Password 


Name 

Email 

Country 

City 


Testing Types 


Testing locations 


Testing 

languages 


|manar 

mana r@ g m a iLco m 


Select your country T 


riydh 

Functional 


Usability 

Localization 

Security 

Mobile 

Saudi Arabia ^ 


ua & : 


Bahrain 

Kuwait 

Qatar 


Arabic 


English 

F ra n ce 


New project 
alert 


Tester 
occupation 
Tester relational 
status 


Health Specialist 


| Single 


Tester parental 
status 

Tester education 


C parent * ‘ not a p arent 

| High school diploma t | 

j Update 


Fig. 13. Tester setting page. 

2. The system will check testers’ validity to receive new announcement alert. The 
system will match the announced project requirements with testers’ preferences 
in order to send customized announcements to interested testers as seen in 

Fig.ua 

3. When validation passed, the new announcement is published in the CST platform 
news page, and an announcement alert is sent to valid testers, as shown in 
Figs. H4l and H5l 


ntact Welcome pm name Sign out 


News list Add new announcement 


Add new announcement 

Project 
Title 

Description 

New Announcement is successfully published Testers notifications have been sent successfully 


project name 1 ▼ 


. _ A 

Publish 


Fig. 14. Published announcement success message. 
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= 

Gmail 


<A 

Search mail ▼ ■■■ 

o 

+ 

Compose 


<- 

Q o ■ (Q © tJ m 1 of 1.051 < > t 


□ 

Inbox 



New project announcement » jmbox * X w 

G3 

★ 

Starred 


# 

CST system 9:03 AM (1 minute ago) -fa 4*. 


e 

Snoozed 


to me - 



Important 



Hi Dear Tester, 


> 

Sent 



There is a new testing project that match your interests. 


& 

Drafts 

20 


The test project announcement is as follow. 

hi dear testers. 


V 

Manar • 

+ 







a new mobile test project is looking for testers from the U S. kindly sign your interest by filling the following 





survey . 






Best, 

CST system 



Fig. 15. Customized announcement email alert. 


9. Stage 5: Final Evaluation 

This section analyzes and evaluates the proposed process models to manage CST 
activities. 

9.1. In-depth interviews 

In-depth interviewing is a qualitative research evaluation method that helps 
researchers to gather much detailed information regarding their thoughts and offers 
an opportunity to explore new issues in-depth in a relaxed atmosphere. Interviews 
are valuable because they are done with domain experts who work in the field; 3 3SH 
The semi-structured interview method was selected, as it offers a space to share 
ideas while keeping the conversation on a specific topic using open-ended questions. 
The process models along with the implemented system were discussed with the 
participants in order to evaluate them and hear their thoughts and suggestions. 

Interviewees’ background 

The interviews involve two companies (BugFinders and Usability Hub) and one 
expert tester. The business of these companies is as follows: 

• BugFinders is a crowdsourced testing platform that delivers several testing solu¬ 
tions such as functional, security, and security testing. The company is rep¬ 
resented by Mr. Luke Edwards, a community manager at BugFinders who 
is responsible for finding testers and monitoring the community in addition 
to solving testers’ problems, whether technical or with a project manager or 
client. 

• UsabilityHub is a crowdsourced testing platform that specializes in usability test¬ 
ing. This company is represented by Mr. Matt Milosavljevic, the CEO and one of 
the founders of UsabilityHub who is responsible for making decisions, handling 
clients, managing employees, and constant development of the company. 
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Table 6. Interviewees, codes and experiences. 


Interviewees Code 

Interviewee Name 

Interviewee Experience 

LI 

Luke Edwards 

2 years in CST and 10 years 



in information technology 

Ml 

Matt Milosavljevic 

4 years in CST and 12 years 



in information technology 

FI 

Fanwell Sibanda 

1 year in CST and more than 5 years 


in information technology 


The crowd tester is represented by 

• Mr. Fanwell Sibanda, a tester at uTest who has worked in many crowdtesting 
projects and has received payment for his work. He works in the information 
systems and business sector and is specialized in information security. 

We contacted several CST companies. However, BugFinders and Usability Hub 
are the two companies that accepted the interview invitation. There were no spe¬ 
cific conditions to select the expert tester except that he/she must have sufficient 
experience of CST (at least one year). Table |6] shows participants’ experience and 
gives a reference code for each. 

Interview findings 

• PI: Announcements alerts 

The interviewees agreed that the proposed process is fine and works well. However, 
LI suggested that a similar approach to recruit specific testers is to have the admin¬ 
istrator add new files to the testers’ profiles and notify them to update them for 
more customized tests. This would reduce the manual work of searching for certain 
testers manually or asking testers manually. 

• P2: Managing n on-active testers 

They agree on the importance of handling non-active testers in order to be fair to 
the client. LI said: “// you promised a certain amount of testers and only a certain 
amount of them is delivered , the client will not get the same value he paid for , which 
is unethical .” They shared a common suggestion that non-active testers should have 
some kind of punishment, e.g. a month’s suspension, demotion, or removal from the 
system after a certain amount of time. Another concern is deciding which testers are 
not active and which testers are testing but simply did not find any bugs to report; 
BugFinders’ solution was to manually monitor testers’ behavior and then make a 
sound decision. On the other hand, FI thinks it is a great idea because there are 
many good testers in the crowdtesting community waiting for a testing chance, and 
testers who could not actively participate should be given second chance because 
life conditions change, and maybe the tester can explain his non-activity. 

• P3: Managing technical issues 

The interviewees shared a common practice. The technical errors are mostly related 
to the clients’ testing environments, and the solution is therefore carried out by the 


2040009-26 





Int. J. Coop. Info. Syst. Downloaded from www.worldscientific.com 

by AUCKLAND UNIVERSITY OF TECHNOLOGY on 05/25/20. Re-use and distribution is strictly not permitted, except for Open Access articles. 


Towards Better Crowdsourced Software Testing Process 


clients. A tester would report a technical error via email; then, after more than one 
tester reports the same issue, the CST platform would stop the test and restart it 
later. They agree that the importance of the proposed solution is that it creates a 
record of communication when the issue is discussed with clients. Moreover, from a 
tester’s point of view, FI said: “ You can wait quite a long period for the response ; 
at some platforms I reached the point of giving up because of late responses , which 
is ineffective. This is why I am very pleased with this activity .” 

• P4: Confirming test reports notifications and managing test project progress 

They agree on having tester payments delayed because of clients’ delayed responses. 
Therefore, alerting clients early on can be a good addition. Additionally, the inter¬ 
viewees agree that this activity would raise both testers’ and clients’ awareness. 

• P5: Managing test project updates 

LI and FI both agree on supporting this activity, sharing that it is a frequent issue 
in crowdtesting projects. LI said: “Very good , I completely agree. We have this 
problem happening all the time. However , the technical solution is not easy , as we 
have thought of a separate application with instant updates , but it would still not 
solve the issue , as testers may not be aware of it while they work ; we also thought 
of text messages for project updates. In our case , when something like this happens 
like an updated project scope , to be fair to the testers we would delete any record of 
bugs outside the scope.” 

9.2. Scenario-based evaluation 

A Scenario-Based Evaluation (SBE) has been conducted as another way of eval¬ 
uating the proposed process models. It gives one the ability to describe concrete 
interactions between different roles and systems.^ Scenarios are scalable and flex¬ 
ible enough to consider the distributed practices over spaces, people, and times 
The suggested evaluation approach provides an assessment through practical sce¬ 
narios conducted with real users. 

To cover the processes mentioned earlier in Table 0 five independent scenarios 
have been created (Appendix A). 

We have written an invitation letter and distributed it through CST forums 
and communities as well as through private messages. We did not reach the same 
participants in Stage 3, though this does not entirely exclude the possibility that 
some of them were involved in both evaluations. 

Twenty domain experts have participated in the evaluation; their information 
are provided in Table 0 The participants did not test the platform together but 
the researchers contacted them individually. We sent a complete explanation of the 
system functionalities and scenarios in advance. For each scenario, we explained 
the pre-conditions (states before the scenario), trigger (the action that initiates the 
scenario) and post-conditions (states after the scenario) to make sure users fully 
understand it then we asked him/her to try the scenario online with the help of the 
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Table 7. Background of the participants. 


Domain Expert 

Role 

Associated Platform(s) 

Experience in CST 

PM1 

Project Manager 

uTest 

2 years, 7 months 

PM2 

Project Manager 

Passbrains 

9 months 

PM3 

Project Manager 

Rainforest 

1 year, 2 months 

CM1 

Community Manager 

BugFinders 

10 months 

CM2 

Community Manager 

Usability Hub 

5 months 

T1 

Tester 

uTest, BugFinders 

2 yrs., 3 months 

T2 

Tester 

uTest, BugFinders 

1 year 

T3 

Tester 

uTest, Passbrains, 
BugFinders 

5 years, 3 months 

T4 

Tester 

Rainforest 

2 years, 4 months 

T5 

Tester 

Usability Hub 

11 months 

T6 

Tester 

Usability Hub 

7 months 

T7 

Tester 

99 tests 

4 years, 3 months 

T8 

Tester 

99 tests 

2 years, 3 months 

T9 

Tester 

uTest 

7 years, 6 months 

T10 

Tester 

uTest 

2 years, 1 month 

Til 

Tester 

uTest 

1 year, 10 months 

T12 

Tester 

uTest 

1 year, 6 months 

T13 

Tester 

uTest 

11 months 

T14 

Tester 

uTest 

8 months 

T15 

Tester 

uTest 

6 months 


Table 8. The evaluation results. 


Process 

Effectiveness Mean 

Standard Deviation 

PI 

4.15 

0.75 

P2 

4.50 

0.69 

P3 

3.70 

0.66 

P4 

4.75 

0.44 

P5 

3.75 

0.64 


researchers. Afterwards, user has entered his/her evaluation of the process under 
test. We aimed in this evaluation to assess the effectiveness of the implemented 
processes. We did not aim to test usability of the system. The evaluation results 
are presented in Table |8] and Fig. HH 

It is clear from the data that most of the processes received positive feedback. 
The processes P4 (Confirm test report notifications and managing test project’s 
progress) and P2 (Managing non-active testers) obtained the highest ratings with 
values 4.75 and 4.50, respectively. The lowest rating mean is for P3 (Managing 
technical issues) with value 3.70, though this value can be still considered good. 


9.3. Discussion 

This research proposed many process models that can help improve CST. In PI 
(announcement alerts), the system uses matching mechanisms between tester and 
project preferences (e.g. location, test type, language, and domain). Although the 
matching mechanisms have been used frequently for assigning suitable testers EHH1 
our findings show that it can also be helpful for determining to which testers we 
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should send new project announcements. The current automatic mechanisms for 
assigning suitable testers helps project managers accelerating the process of assign¬ 
ment since there could be large number of candidate testers. Similarly, automating 
the process of sending announcements will help testers recognize which test project 
announcements are relevant to them (i.e. in utests, there are 250,000 testers, per¬ 
haps minority is interested in specific types of testing such as security testing). We 
believe that current CST companies have not concentrated enough on tester’s chal¬ 
lenges associated with their daily use of the platforms, though the implementation 
of this process is not difficult and could save the time of testers. 

In P2 (managing non-active testers), the novelty is in thinking of a new approach 
to stop losing test slots as a result of having testers promise to participate in a 
test project then disappear. This approach aligns with the call in the literature 
towards monitoring progress of testers during test cycles to maximize the bene- 
fitsJ 291431 Although the proposed approach can help replace the manual process of 
monitoring tester’s behavior (such as the one used by BugFinders), it has the dis¬ 
advantage that it does not differentiate between testers who are not active and 
those who are working on testing but did not find any bugs to report. There will 
be a slight overhead for the latter since they will need to respond to the automatic 
message to confirm their participation. 

Regarding P3 (managing technical issues), the system links the technical issues 
with testing projects creating a repository of technical issues accessible at any time. 
The email solution to report and manage technical issues currently used by the two 
companies BugFinders and UsabilityHub would be more problematic if they get 
larger number of projects at the same period of time because email systems are not 
specialized to keep track of such cases. 

Further, P4 (confirming test report notifications and managing test project’s 
progress) and P5 (managing test project updates) supports speeding up the CST 
process through notifying clients to confirm bugs early in the bug’s evaluation activ¬ 
ity and also informing testers of project progress in order to avoid any additional 
unnecessary work. 
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Table 9. Technical contributions and associated challenges and improvements. 


Process Main Technical Contributions Possible Challenges/Improvements 

Model 


PI - System matches the announced 

project requirements with testers’ 
capabilities/preferences (e.g. loca¬ 
tion, test type, language, and 
domain) 

P2 - System automatically creates a 

timer to check non-active testers in 
the middle of each test cycle 

- System detects non-active testers 
(i.e. who did not submit any test 
reports) and notifies them. 

- System automatically discards 
those who did not respond from the 
test team and inform other selected 
testers who did not have the chance 
to book a test slot of new open test 
slot 


P3 - System links technical issues with 
projects 

- System coordinates sending techni¬ 
cal issues to the right PM 

P4 - System automatically sends a noti¬ 
fication to client when a PM evalu¬ 
ates a bug and a decision is needed 
from client 

- When the required number of bugs 
is reached (i.e. confirmed by client), 
the system will inform testers 

P5 - System links each project update 
with the corresponding project 

- System sends an in-application 
new notification to all testers work¬ 
ing on the related project 


- System uses only simple matching 

mechanism. Advanced recommendations 
can be developed using mining techniques 


- We considered the middle of test cycle as 
checkpoint of tester’s activity as 
recommended by the interviewees. 
However, further research may be needed 
for best determination (i.e. factors such 
as the length of test cycle could be 
considered) 

- System does not differentiate between 
testers who are not active and those who 
are working on testing but did not find 
any bugs. Further improvement can be 
made to distinguish between the two 
kinds of testers (e.g. through login history 
or the access to the system under test.) 

- The new tester will not take the full 
chance as those who started the test 
cycle from the beginning. This needs to 
be considered during evaluating tester’s 
performance 

- Further support can be provided through 
mining the repository to find similar 
technical issues and suggesting them to 
testers and/or PMs 

- A possible challenge occurs if client is not 
responsive or not willing to evaluate test 
reports individually. This risk can be 
mitigated by raising awareness about the 
benefits of this process in accelerating 
test cycle 


The proposed improvements are with diverse difficulty level. While some seem 
relatively straightforward, others may require deeper understanding to the CST 
process such as P2 (managing non-active testers). Table |9] analyzes the technical 
contributions included in each process model. It also discusses the potential chal¬ 
lenges associated with applying these processes and provides suggestions for further 
improvements. As can be seen in the table, the current implementation provides 
basic technical contributions that are mostly easy integrated into the current CST 
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platforms. However, we critically evaluate these contributions through thinking of 
what possibly affect their use and how to improve the processes further. 


10. Conclusion 

This paper supports the effective management of CST by proposing a set of pro¬ 
cess models that aims to overcome the current limitations. This research shows that 
CST is still in need for further support. We have identified eight possible improve¬ 
ments, among them six have been developed. The evaluation shows that these 
improvements are sound and can add value to CST. This research contributes in 
promoting CST practice which ultimately would help support producing better test 
quality. 

Our findings suggest that the current CST platforms are still less mature to 
manage the CST process and that there is a space for improvement. This is perhaps 
due to the fact that the practice of crowdsourcing software testing activities is 
relatively young as most CST companies were established in the current decade. 
We believe that with the expected growing size of the business of these companies 
over time, it would be more difficult to rely on ad-hoc solutions to manage their 
process. 

Several enhancements and extensions can be further applied to the work pro¬ 
vided in this research as discussed in Table |9l For example, the non-active testers’ 
detection methods can be improved to be more efficient and sophisticated using 
advanced technologies to monitor user behavior. Additionally, an automatic report¬ 
ing system based on the technical error messages repository can be developed; 
these reports will help identify common technical issues and their areas within test 
projects in order to avoid them in the future. 
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Appendix A 

Scenario 1: Announce a new test project 

Pre-conditions: A test project has been submitted by a client and discussed with 
a project manager. 

Trigger: Project manager initiates writing new announcement. 

Post-conditions: The new announcement is posted on the platform website and 
an email is sent. 
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Steps: 

1. The project manager writes a new post and publishes it on the CST platform 
news page. 

2. The system checks and matches test project requirements with testers’ prefer¬ 
ences, and then sends an alert only to testers interested in this kind of project 
(location, language, test type). 

Scenario 2: Manage non-active testers 

Pre-conditions: A test cycle has started and testers are able to test and submit 
bugs. 

Triggers: Non-active testers are discovered. 

Post-conditions: Test cycle ends and testers’ evaluations are updated. 

Steps: 

1. At the middle of the test cycle, the system checks all testers’ participation in 
this test cycle. 

2. The system finds a tester that has accepted this test cycle but did not participate 
for some reason. 

3. The system sends an email alert to that tester to ask for active participation 
and to inform them of automatic withdrawal from testing after a certain period. 
There are two possible scenarios, depending on the tester’s response: 

a. Tester confirms his intent to participate in this test cycle, and the system 
keeps him on the team. 

b. Tester ignores the email, and after a certain period of time has passed, the 
system drops the tester from testing team, and informs the project manager 
and other accepted testers of an open test slot. 

Scenario 3: Report a technical error message 

Pre-conditions: A test cycle has started and testers should start testing. 

Triggers: Tester tries to open the test environment but cannot due to a technical 
issue. 

Post-conditions: The technical error message is answered and the tester can start 
testing. 

Steps: 

1. Tester tries to access testing environment and start testing. 

2. Tester faces a technical issue (e.g. installing phone app, wrong credentials,...). 

3. Tester goes back to the CST system and makes new technical error request by 
filling out all required information and submitting. 

4. The system sends an alert to the project manager of this test project. 
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5. The project manager reviews the submitted request and suggests a suitable 
solution. 

6. The system informs the tester of new update regarding the reported technical 
issue. 

7. All transactions are saved in the system for future reference. 

Scenario 4: Confirm test report notifications and managing test project’s 
progress 

Pre-conditions: A test cycle has started and testers have started submitting bugs. 
Triggers: The project manager starts validating bugs. 

Post-conditions: The test cycle ends, and testers/clients are notified. 

Steps: 

1. The project manager starts validating submitted bugs. 

2. Once a bug is validated by the PM, the system automatically sends an email to 
the client to confirm the evaluated bug for faster processing. 

3. The project manager continues validating bugs until the required number of bugs 
is reached. 

4. The system sends an email to both client and testers informing them of the bug’s 
evaluation progress, so no additional work is wasted. 

5. The system automatically informs testers of the end of a test cycle due to reach¬ 
ing end time or the number of accepted bugs. 

Scenario 5: Updated project scope 

Pre-conditions: A test cycle has started and testers are able to test and submit 
bugs. 

Triggers: The client informs his CST project manager of new information regarding 
the test project. 

Post-conditions: Testers receive an email alert with a new test project update. 

Steps: 

1. The client informs the project manager of new test project scope. 

2. The project manager updates test project details. 

3. The system sends an email alert to testers regarding new updates. 

4. The system displays an in-application notification for testers regarding new 
updates. 
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